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Today Challenge: Economical 


Average PUE of your largest data center: 
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Source : Uptime Institute survey of over 1100 data centers 


N 


Total data center power ( kw ) 


Data center Power usage effectiveness(PUE)= 
Total IT Power ( kw 
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A Shehabi: LAWRENCE BERKELEY NATIONAL LABORATORY "United States Data Center Energy 
Usage Report", 2016. 


e The energy consumption data center for cooling is quite large. 
e How to achieve low PUE and optimized TCO? That has become a new challenge. 


D@LLEMC 


A new Challenge: Technical 


Rack with 42*1U Server power trend 


Every 1 kiloWatt (kW) of rack power needs 1kW 50000 
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Full Air-Cooling is not going to meet the Year of Product Announcement 
heat dissipation demand any more Datacom Equipment Power Trends and Cooling Applications --ASHRAE 


D@LLEMC 


TDP trends 


GPU Vs CPU TDP Trend 
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Source : Alibaba.com - Immersion cooling for Green Computing - OCP2018 
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Technology trends 
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Figure 1.8: Approximate distribution of peak power usage by hardware subsystem in a modern data 
center using late 2017 generation servers. The figure assumes two-socket x86 servers and 12 DIMMs 


per server, and an average utilization of 80%. 


Source : Barroso, Holzle, Ranghanathan - The Datacenter as a Computer - M&C 
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Example breakdown of peak power usage of a datacenter using 2012 generation servers. Assumes 
two-socket x86 servers, 16 DIMMs and 8 disk drives per server, and an average utilization of 80% 


Chung-Ta King - Department of Computer Science - WAREHOUSE-SCALE 


COMPUTERS, National Tsing Hua University, Taiwan DALL E MC 


Increasing Efficiency 


Efficiency increase = Reduction in Total Cooling Power consumption. Where to act ? 


Compressors/Heat Exchangers (better HW, AT) 

Pumps (reducing speed/ reducing mass / eliminating) 

Fans (better HW, reducing speed / eliminating) 

Better coolant media (better heat carrying capacity / better fluid temperature) 
Better air flow paths (compartimentazione) 


As heat density increases air becomes a less efficient coolant 


Liquid cooling provides a mean in which thermal resistance can be reduced dramatically 


Principle: Qioap = MCpAT = rVCpAT 
Water has about 3.300 times higher heat carrying capacity than air! 


Rule of thumb: the more the cooling liquid comes near the heat source, the better the efficiency is 
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Liquid Cooling Landscape 


68 F 
CHILLED WATER 


Do. Coupled i 
Air is still the only mean to | 
e Close Coupled Cooling cool board/chip 
e Limited by existing chip ROOM OUT 
—  |In-Row’, In-Rack, Rear Door Exchangers maximum temps 


Requires additional fans 


Cold Plates 
e DLC (Cold Plates) e Individual Heatsinks 
e Board-Specific 
— Positive Pressure, Negative Pressure e Individual Chip Fluid- 
cooling 


Immersion 

7 . e Specialty fluids (SSS) 

e Immersion Cooling * Can Require separate 
Cooling Coil 


7 Single Phase, Two-Phase e Orientation sensitive 
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e Open-Loop Solution (air flows interact with 
the ambient room environment. ) 


e Row-Based Air Conditioning units are installed 
inside the rack rows. 


e Heat removal of up to 70 kilowatts per unit 
e Controlled in-row cooling 

e Row air containment 

e Modularity 

e Typical Inlet water temperature 12-23°C 


e Air flows generally follow short and linear 
paths, reducing, in this way, the necessary 
power needed to rotate the fans and 
increasing the energy efficiency. 


Heat Exchanger and Fan Front View Rear View 
Assembly 


Images: www.apc.com 


In-Rack Cooling 


e Closed-loop Solution (no interaction with the 
ambient room environment). 


e The cooling system is joined to the server rack and 
both of them are completely sealed; the solid doors 
on the enclosure and In-Row Air Conditioners 
contain the air flow, directing cold air to the server 
inlet and exhaust air, by using fans, through the 
cooling coil. 


e Heat removal of up to 30-35 kilowatts per unit 
e One unit can serve up to two racks (up to 48U) 


e Typical Inlet water temperature 12-23°C 


e Right Equipment orientation (ie front-to-back) in the 
rack and proper cabling and positioning a must! 


Rear-Door Heat Exchangers 


65° F 
CHILLED WATER 


. x i . RETURNING TO CHILLER AT 
e Open-Loop Solution (air flows interact with the 75° F 
ambient room environment. ) 


STANDARD OR CUSTOM 
SERVER RACK 


e This type of solution is based on the substitution rors 


of the rear door of an existing rack 


75° F 


e Heat removal of up to 80 kilowatts per unit ROOM AIR IN 


e One unit per rack (up to 48U) ROOM AIR OUT 
e Rack adapters available for most standard racks 


e Typical Inlet water temperature 12-23°C 
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ExpLopeD View oF Active Rear Door 
Rack CooLina SYSTEM 


Why Direct Liquid Cooling? 


e Heat density of servers is getting greater. 
e Getting sufficient air flow in the confined space within servers is getting harder 
e The latest Intel CPUs are > 200W & some cannot be air cooled 


e New systems design do accept a higher thermal delta between Inlet and Outlet air temperatures of 
equipment (ASHRAE TC 9.9). This can lead to «normal» Tout of about 60°C (!) 


- Consequences for cables, optics and other components 
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DLC Components 


e DLC or Direct Liquid Cooling, also known as Cold Plate, uses a hollow metallic 
connection to the parts of a server, such as CPU or memory. Through this hollow 
metallic link, liquid flows taking the heat away within the server chassis. Coldplate 
assemblies replace heatsinks and are purpose-designed 


e Connection to these heatsinks is via dry-break couplings, exiting the rear of the 
server, thereby permitting connection & disconnection with no leaks. A manifold 
within the rack manages & merges the water connections. Rack manifolds are made 
with reliable stainless steel and typically occupy a PDU mounting location at the rear 
of the rack. 


e Heat Exchanger Modules connect to the rack manifold to pump coolant to the racks 
and exchange heat from the servers with facility water. 


e Solution sold and supported by Dell on C6420 
e Heat removal of up to 80 kilowatts per rack (!). 


e Typical Inlet water temperature 27-45°C 


DLC: Pros 


e Wider Cooling Margin allows higher water temperatures, which 
means less chiller infrastructure. 


e Also allows heat extraction & reuse from the outlet water (i.e. 
through building heating system). 


e High Temperature water allows for more hours of free cooling 
through low-maintenance dry coolers 


e EERincreasing with Inlet water temperature increase 


e Enables lower TCO costs with more efficient cooling 
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Source : R.Ricci - Infrastruttura: come cambia il clima in sala calcolo - INFN CCR Workshop 2018 
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water-cooled cold plate 


Mandata 7°C) — Mandata 12°C 


Mandata a 20 
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DLC: Cons 


° Only a portion (up to 80%) of the total generated heat is removed with liquid, 


fans still required. 


° A significant portion (about 20%) of the total DLC system investment is lost 


when servers are replaced (coldplate assemblies and tubing are purpose- 
designed) 


° No warranty that different server vendors will adopt the same DLC system in 


the future (quite all enterprise servers manufacturer are however converging 
toward Staubli push-fit connectors, at today) 


° High water temperatures (usually) require specific cooling circuits 
° Proper Water Quality required (ASHRAE-D-90564) 


° Using a single cooling system for traditional cooling and DLC may not make 


sense, as the cooling fluids would usually be at different temperatures. 


_ But output water from Rear Door Heat Exchangers could be reused as inlet water 
for DLC (if DC cooling system can support this) 
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70% heat capture 
into Direct Liquid 
Cooling 
(CPU Coldplates) 
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Generated by 
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30% heat capture 
into Indirect Liquid 
Cooling 
(Rear Door) 
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Immersion Cooling 


e Immersion cooling is the practice of submerging computer 
components or systems in a thermally, but not electrically, 
conductive liquid (dielectric coolant) such as mineral oil, 
fluorocarbon-based fluids, synthetic fluids. 


e In traditional cooling systems, heat must transfer through 
several levels of thermal interface materials, air, heat 
exchangers and working fluids. Immersion cooling simplifies 
thermal design and increases heat transfer efficiency. 


e Depending on the properties of the coolant fluids, we can 
classify the Immersion Cooling techniques in: 
= Single-Phase Immersion Cooling 
= Two-Phase Immersion Cooling 


Single-Phase 


e In single-phase immersion cooling the fluid has a 
higher boiling point and remains in its liquid phase 
throughout the process. 


e Electronic components are submerged in a non- 
conductive liquid bath in an accessible, sealed 
enclosure 


e The heat from the chip is transferred to the fluid 


e Pumps are often used to flow the heated fluid to a 
heat exchanger, where it is cooled and cycled back 
into the bath 


e Heat removal of up to hundreds of KW per unit 


e Environment-Friendly Fluids (even cooking oil 
concepts! ©) 


Single-Phase High-level breakdown 


Server Pod 


Pump & Filter Unit 


Warm oil output 


Cold oil input 


Cold water input 


Two-Phase 


e Two-phase passive immersion cooling exponentially 
increases heat transfer efficiency from boiling and 


condensation of cooling fluids. The Passive 2-Phase 


Immersion Cooling Cycle 


Vapor condenses on coil 
or lid condenser 


e Electronic components are submerged in a non-conductive 
liquid bath in an accessible, sealed enclosure 


Fluid recirculates 
passively to bath 


e The generated heat boils fluid into a gaseous state and it 
rises to the top of the enclosure 


Vapor rises to top 


e The gas condenses on the lid or a condensation coil and 
rains back into the bath to repeat the cycle 


Heat generated on chip 


e Heat transfer efficiencies can exceed boiling water in a sai fee clive ni ae 


material that can safely come in direct contact with 
electronics (High latent heat in liquid vaporization) 
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e Heat removal of up to hundreds of KW per unit 


Immersion Cooling: Challenges 


Material Compatibility 


— Several studies demonstrated coolant fluid pollution due to dissolved chemicals coming from cables 
and HW/chips/plastic containers 


Signal Integrity 
— PCB strips designed for copper-in-the-air signal transmission (dielectric fluid affecting EM properties) 


Reliability 
—  Micro-cavitation in Two-Phase; Long-Term effects on electronics not well known 


Optics and Optical devices 
— Refraction Index of coolant fluid very different from air! 


IT Equipment Serviceability 
Environmental Footprint (Global Warming Potential) 


Thermal Resistance 
DLC may be far better of immersion in cooling liquids! (We’ve found 0,05 °C/W vs 0,2 °C/W!) 


DALL 


EMC 


Top 500: Cooling Methods 


Cooling solutions for 
supercomputers 


Air cooling 

Water cooling 

One-phase immersion cooling 
Two-phase immersion cooling 


Cooling solutions applied in Top50 
Supercomputing systems [1] 


Air 
Cooling 


Water 
Cooling 


One-phase 
immersion 
cooling 


Where is two-phase immersion cooling? 


[1] “Top 500 supercomputers,” Top 500 list as of November 2017. 


www.top500.org 


21 


Conclusions 


° Close Coupled Cooling is still the best solution, if Racks TDP falls into specifications 
Full Investment recovery at IT HW replacement 


Very long service life (independent from IT HW) 


Very Low maintenance costs 


Efficiency quite similar to Hybrid DLC solutions 
TCO balanced by minor CAPEX investment at start 
Rear Door Exchangers may remove the same heat qty as DLC 


° DLC to be used only when Close Coupled not sufficient, not applicable, or in new systems 
About 20% of the DLC solution investment is lost at IT HW replacement 


Standard air cooling solution still required 


Difficult implementation within existing cooling solutions (cooling fluids at different temperatures) 


° Immersion cooling still a niche solution and, at today, practically unapplicable for standard DC IT 


Higher PUE and kW/rack values, however: 


> 


> 


> 


> 


Supportability, Environmental and Technological concerns 
Needs specific support infrastructure (because of weight, exchangers etc) 
Exciting PUE, however TCO reduction unclear 


TCO influenced by replacement fluid costs. 
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Thank you 
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